R115777 [(B)-6-[amino(4-chlorophenyl)(1-methyl-1H-imidazol-5-yl)-methyl]-4-(3-chlorophenyl)-1-methyl-2(1H)-quinolinone] is a potent and selective inhibitor of farnesyl protein transferase with significant antitumor effects in vivo subsequent to oral administration in mice. In vitro, using isolated human farnesyl protein transferase, R115777 competitively inhibited the farnesylation of lamin B and K-RasB peptide substrates, with IC50s of 0.86 nM and 7.9 nM, respectively. In a panel of 53 human tumor cell lines tested for growth inhibition, approximately 75% were found to be sensitive to R115777. The majority of sensitive cell lines had a wild-type ras gene. Tumor cell lines bearing H-ras or N-ras mutations were among the most sensitive of the cell lines tested, with responses observed at nanomolar concentrations of R115777. Tumor cell lines bearing mutant K-ras genes required higher concentrations for inhibition of cell growth, with 50% of the cell lines resistant to R115777 up to concentrations of 500 nM. Inhibition of H-Ras, N-Ras, and lamin B protein processing was observed at concentrations of R115777 that inhibited cell proliferation. However, inhibition of K-RasB protein-processing could not be detected. Oral administration b.i.d. of R115777 to nude mice bearing s.c. tumors at doses ranging from 6.25-100 mg/kg inhibited the growth of tumors bearing mutant H-ras, mutant K-ras, and wild-type ras genes. Histological evaluations revealed heterogeneity in tumor responses to R115777. In LoVo human colon tumors, treatment with R115777 produced a prominent antiangiogenic response. In CAPAN-2 human pancreatic tumors, an antiproilferative response predominated, whereas in C32 human melanoma, marked induction of apoptosis was observed. The heterogeneity of histological changes associated with antitumor effects suggested that R115777, and possibly farnesyl protein transferase inhibitors as a class, alter processes of transformation related to tumor-host interactions in addition to inhibiting tumor-cell proliferation.